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Abstract 

This  paper  addresses  the  issue  of  how  to  make  the  task  of  writing  correct  and  ef¬ 
ficient  parallel  programs  easier.  Domain  morphism  is  a  new  construct  for  specifying 
parallel-program  optimization  at  a  high  level,  and  enables  an  equation  al  program 
transformation  whereby  the  optimized  program  containing  implementation  details 
is  derived  mechanically.  In  addition,  we  introduce  the  constructs  index  domain, 
data  field,  communication  form,  and  hyper-surfaces  for  facilitating  parallel  pro¬ 
gramming  and  generating  eflficient  target  parallel  programs  with  explicit  synchro¬ 
nization  and  communication.  We  describe  an  equational  theory  and  new  program 
transformation  procedures  motivated  by  these  new  constructs.  A  programming 
example  illustrating  the  use  of  these  constructs  and  the  program  transformation 
steps  is  given. 
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1  Introduction 

Writing  correct  programs  is  not  easy;  making  programs  run  efficiently  requires  further  com¬ 
plex  process  of  matching  the  logical  structure  of  the  program  to  the  underlying  physical 
structure  of  the  machine.  With  the  arrival  of  large-scale  multi-processor  computers,  we  face 
the  challenge  of  parallel  programming  where  the  allocation  of  processors  and  the  cost  of 
inter-processor  communication  now  enter  into  the  question  of  efficiency. 

This  paper  addresses  the  issue  of  how  to  make  the  task  of  writing  correct  and  efficient 
parallel  programs  easier.  Our  approach  to  programming  starts  with  a  high-level  problem 
specification,  through  a  sequence  of  optimizations  tuned  for  particular  parallel  machines, 
leading  to  the  generation  of  efficient  target  code  with  explicit  synchronization  and  commu¬ 
nication.  The  key  new  concept  of  domain  morphism  specifies  the  transformation  from  one 
program  to  the  next  in  the  sequence.  It  plays  a  dual  role,  first,  as  a  construct  for  specify¬ 
ing  an  optimization  strategy,  and  second,  in  enabling  an  equational  program  transformation 
whereby  the  optimized  program  can  be  derived  mechanically. 

Domain  morphisms  capture,  as  mathematical  objects,  the  essence  of  existing  loop  trans¬ 
formations  in  vectorizing  and  parallelizing  compilers,  as  well  as  new  techniques  of  data  lay¬ 
out  and  efficient  inter-processor  communication  for  distributed  and  shared-memory  machines 
with  memory  hierarchy. 

Formalizing  the  optimizations  a  parallelizing  compiler  must  do  requires  the  notion  of 
domain  morphism,  which  in  turn  requires  an  equational  theory  of  programs.  We  have  found 
the  traditional  program  transformations  inadequate  in  that  operators  such  as  folding  and 
unfolding  [?,  ?,  ?])  are  applied  only  to  expressions,  whose  extensions  remain  unchanged. 
The  equational  theory  allows  us  to  derive  a  new  program  which  is  equivalent,  in  a  precise 
way  defined  by  the  domain  morphism,  but  which  is  extensionally  different  from  the  original 
in  that  it  is  defined  over  the  new  domain. 

The  rest  of  the  paper  is  organized  as  follows.  In  Section  ??,  we  introduce  the  domain 
morphism  and  other  constructs  for  specifying  peirallel  program  transformations.  Section  ?? 
introduces  the  equational  theory  for  program  transformation.  The  next  section  presents  in 
full  detail  an  example  of  domain  morphism  induced  program  transformation  on  a  program 
for  solving  Laplace’s  equation  using  successive  over-releixation  (SOR). 

2  New  Constructs 

In  the  following,  we  introduce  a  few  programming  constructs  which  are  important  for  par¬ 
allel  programming,  where  of  central  importance  is  the  notion  of  domain  morphism.  Index 
domains  are  abstractions  of  the  “shapes”  of  composite  data  structures,  which  in  most  cur¬ 
rent  programming  languages  are  not  first-class  objects.  Data  fields  generalize  the  notion 
of  distributed  data  structures,  unifying  the  conventional  notions  of  arrays  and  functions. 
Communication  forms  defined  over  an  index  domain  are  means  of  specifying  the  data  depen¬ 
dencies,  and  the  inverse  of  a  communication  form  (defined  precisely  later)  provides  useful 
directives  for  optimizing  inter-processor  communication.  Hyper-surfaces  are  for  specifying 
the  boundaries  of  index  dom2dns.  Finally,  domain  morphisms  describe  the  reshaping  of  in¬ 
dex  domains  aimed  at  optimizing  data  or  control  structures  for  efficiency  reasons  as  well  as 
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the  necessary  mapping  from  the  logical  structure  of  the  problem  to  the  physical  domain  of 
machine  and  sequencing  in  time. 


Notation  A  function  /  from  domain  A  to  codomain  B  is  denoted  f  :  A  B,  and  is  defined 
using  function-abstraction:  /  =  fn(x):>l{T[x]}.  The  conditional  construct  is  represented  in  a 


two-dimensional  form 


and  its  meaning  is  h,  if  bi  is  true  for  one  i,  and  undefined 


otherwise. 


2.1  Index  Domains 

An  index  domain  D  consists  of  a  set  of  elements  (called  indices),  a  set  of  functions  from  D  to 
D  (called  communication  operators),  a  set  of  predicates,  and  communication  cost  associated 
with  each  communication  operator. 

In  essence,  an  index  domain  is  a  data  type  with  communication  cost  associated  with  each 
function  or  operator.  The  reason  for  making  the  distinction  is  that  index  domains  will  usually 
be  finite  and  they  are  used  in  defining  distributed  data  structures  (as  functions  over  some 
index  domain),  rather  than  their  elements  being  used  as  values.  For  example,  rectangular 
arrays  can  be  considered  to  be  functions  over  an  index  domain  consisting  of  a  set  of  ordered 
pairs  on  a  rectangular  grid.  Also,  the  elements  of  an  index  domain  can  be  interpreted  as 
locations  in  a  logical  or  real  space  and  time  over  which  the  parallel  computation  is  defined. 
So  we  classify  index  domains  into  certain  kinds  (second  order  types)  according  to  how  they 
are  to  be  interpreted  (for  example,  eis  time  or  space  coordinates). 

Basic  Index  Domains  We  now  give  examples  of  basic  classes  of  index  domains.  The 
most  often  used  are  the  interval  and  the  hypercube  domains: 

An  interval  domain,  denoted  interval(m,n),  where  m  £ind  n  are  integers  and  m  <  n,  is 
an  index  domain  whose  elements  are  the  set  of  integers  {m,m-|-l,m-l-2,  ...,n}  with  the 
usual  integer  functions  and  predicates.  The  communication  operators  are  prev  :  i  i  —  1 
and  next :  i  i  -f  1,  with  communication  cost  1.  The  operators  lb  and  ub  return  the  lower 
bound  (m)  and  the  upper  bound  (n)  of  the  interval  domain  respectively.  When  m  >  n,  we 
define  the  index  domain  to  be  the  same  except  that  prev  and  next  have  reversed  meaning. 

A  hypercube  domain  of  dimension  n,  denoted  hcube(n),  is  an  index  domain  with  2"  ele¬ 
ments  of  the  form  (xo, •  •  .,x„_i),  where  each  x,-  is  either  left  or  right  (or  just  0  or  1),  and 
communication  operators  hcnb(j,  k),  for  j  an  element  of  the  data  type  and  0  <  k  <  n  —  I, 
which  maps  the  element  j  to  its  neighbor  along  the  kih  dimension,  each  with  unit  commu¬ 
nication  cost.  Predicates  are  \ehl{k)  and  right?(fc),  for  testing  whether  an  element  is  on  the 
left  or  right  half  of  the  fcth  dimension. 

Index  Domain  Constructors  Given  index  dom£iins  D  and  E,  we  can  construct  their 
product  {D  X  E),  disjoint  union  (coproduct)  (D  -f  E),  and  function  space  D  E,  in  the 
usual  way. 
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2.2  Kinds  of  Index  Domains 

As  discussed  above,  it  is  useful  to  indicate  how  an  index  domain  is  to  be  interpreted  by 
“typing”  index  domains  with  “kinds”  (second-order  types).  Analogous  to  the  first-order 
typing,  D\K]  will  mean  that  the  index  domain  D  is  of  kind  K.  The  following  are  the  kinds 
we  have  found  useful  for  compiler  optimizations; 

Universal  (W):  the  kind  of  any  index  domain  as  well  as  any  other  data  type,  such  as  the 
integers  and  the  reals. 

Temporal  (T):  the  kind  of  index  domain  representing  time  coordinates  (subkind  of  U). 
Spatial  (5):  the  kind  of  index  domain  representing  space  coordinates  (subkind  of  U). 
Processor  (P):  the  kind  of  index  domain  representing  processor  coordinates  (subkind  of 

‘5). 

Memory  {M):  the  kind  of  index  domain  representing  memory  locations  within  a  processor 
(subkind  of  «S).  Memory  hierarchy  can  be  introduced  with  other  kinds  Af,  which  are 
subkinds  of  Al,  where  i  ranges  over  the  levels  of  the  memory  hierarchy. 

2.3  Index  Domain  Morphisms 

Index  domain  morphisms  formalize  the  notion  of  transforming  one  index  domain  into  an¬ 
other,  with  the  kinds  of  the  domains  indicating  the  change  of  interpretations. 

Definition  Let  D  and  E  be  two  index  domains.  An  index  domain  morphism  is  a  function 
g  from  D  to  E  such  that  for  all  elements  x  and  y  in  D,  if  there  exists  a  composition  t  of 
communication  operators  tiot20'  •  •  r*  over  D  such  that  y  =  t(x),  then  there  is  a  composition 
t'  of  communication  operators  o  Tj  o  •  •  •  T|(,  over  E  such  that  g{y)  =  t'(^(x)). 

The  extra  constraint  is  to  ensure  that  if  there  is  a  path  in  the  first  domain  from  x  to  y, 
then  there  is  a  path  from  g{x)  to  g{y). 

Reshape  Morphisms  For  any  index  domain  morphism  g  :  D  E,  Si  left  inverse,  if  it 
exists,  is  an  index  domain  morphism  h  :  E  D  such  that  hog  =  l^j,  the  identity  morphism 
over  D.  If  5  is  also  a  left  inverse  of  h  {i.e.,  g  oh  =  l^),  then  h  is  called  the  inverse  of  g  and 
is  denoted  by  A  morphism  that  has  an  inverse  is  commonly  known  as  an  isomorphism 
but  here  will  be  called  a  reshape  morphism  to  emphasize  the  idea  of  “reshaping”  one  index 
domain  into  another. 

To  require  the  existence  of  the  left  inverse  implies  that  a  reshape  morphism  must  be  bijec- 
tive.  However,  we  can  easily  derive  a  reshape  morphism  from  an  injective  domain  morphism 
by  restricting  the  codomain  as  follows:  Given  an  injective  domain  morphism  g'  :  D  —*  E, 
the  image  of  D  under  g',  denoted  image(Z?,5'),  is  an  index  domain  whose  elements  are  the 
image  of  the  elements  of  D  and  whose  communication  operators  au-e  those  of  E.  Clearly, 
g  •.  D  —*  image(D,y'),  derived  from  g',  now  has  a  well-defined  left  inverse. 

Here  are  examples  of  some  useful  reshape  morphisms: 
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•  An  affine  morphism  is  a  reshape  morphism  which  is  an  affine  function  from  one  prod¬ 

uct  of  intervals  to  another.  Affine  morphisms  unify  all  types  of  loop  transformations 
(interchange,  permutation,  skewing)  [?,  ?,  ?,  ?,  ?],  and  those  for  deriving  systolic  al¬ 
gorithms  [?,  ?,  ?,  ?].  For  example,  if  Di  =  interval(0, 3)  and  =  interval(0,6),  then 
g  =  fn(i,  j)  :  Di  x  D2{(j,i)  :  x  }  is  an  affine  morphism  which  effectively  performs 

a  loop  interchange. 

Another  example  illustrates  a  slightly  more  interesting  codomain  E  of  the  morphism  g 
by  taking  the  image  of  a  function  g'. 

Dq  =  interval(0, 3)  D  =  DqY.  Dq 
E  =  image(D,5')  wKere{  5' =  fn(i,j):Z){(i  -  j,t -1- j)}  } 

9  =  -i,i -}- 1):£}  5“^  =  fn(i,  j):£{((i -|- j)/2,(;  -  i)/2):D} 

Whenever  it  is  legal  to  apply  this  affine  morphism  to  a  2-level  nested  loop  structure — 
consistent  with  the  data  dependencies  in  the  loop  body  [?,  ?,  ?,  ?,  ?,  ?]) —  a  similar 
structure,  but  “skewed”  from  the  original,  is  generated.  The  most  common  case  is 
when  elements  of  the  inner  loop  can  be  executed  in  parallel,  but  where  only  half  of 
the  elements  are  active  in  each  iteration  of  the  outer  loop.  In  this  example,  the  index 
domain  E  has  holes,  and  so  guards  in  the  loops  must  test  whether  i  +  j  and  i  —  j  are 
even,  since  only  these  points  correspond  to  the  integral  points  in  D 

In  Section  ??,  we  will  show  in  detail  how  such  a  transformed  loop  is  derived  algebraically. 

•  A  uniform  partition  of  a  domain  D  is  a  reshape  morphism  g  :  D  —*  Di  x  D2,  with 
the  property  that  the  codomain  has  a  greater  number  of  component  domains  (i.e.,  is  of 
higher  dimensionality). 

For  example,  U  D  —  interval(0, 11),  Di  =  interval(0,3),  and  D2  =  interval(0, 2),  then 

5  =  fn  t  :  Z)[5]{(t/3,t  mod  3)  :  Di[P]  x  D2[M]} 

is  a  uniform  partition  that  distributes  the  elements  of  an  index  domain  of  the  spatieil 
kind  as  follows:  the  12  elements  are  partitioned  into  3  blocks  of  4  elements  each,  each 
block  is  cissigned  to  a  processor,  and  each  element  in  a  block  is  assigned  to  some  memory 
location. 

In  general,  a  partition  is  a  reshape  morphism  g  :  D  J2{i:I)  D{i),  where  /  is  some 
index  domain  and  denotes  generalized  sum  over  all  the  Z)(t)’s.  The  idea  is  that  the 
domain  D  is  partitioned  into  disjoint  union  of  subdomains  Z)(t),  which  may  contain 
different  number  of  elements. 

There  are  numerous  other  forms  of  reshape  morphisms  ranging  from  “piece-wise  affine” 
morphisms  for  more  complex  loop  transformations  [?],  to  those  that  are  mutually  recursive 
with  the  program  (to  be  transformed)  for  dynamic  data  distribution. 

Refinement  Morphisms  A  domain  morphism  that  does  not  have  an  inverse  is  called 
refinement  morphism.  Though  we  shall  not  deal  with  them  in  this  paper,  refinement  mor¬ 
phisms  are  useful  in  representing  more  complex  relationships  between  domains,  such  as  the 
transformation  of  one-to-many  broadcasts  into  binary  tree  bro2Micasting  [?]. 
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2.4  Data  Fields  and  Data  Field  Morphisms 

Definition  A  data  field  is  a  function  over  some  index  domain  D  into  some  domain  of  values 

V. 

Usually,  V  will  be  the  integers  or  the  floating  point  numbers;  however,  for  higher-order 
data  fields,  it  can  be  some  domain  of  data  fields.  Data  fields  unify  the  notion  of  distributed 
data  structures,  such  as  arrays,  and  functions.  A  parallel  computation  is  specified  by  a  set 
of  data  field  definitions,  which  may  be  mutuaJly  recursive. 

To  illustrate  the  use  of  data  fields,  consider  the  following  program  segment  written  in 
some  imperative  language  (assuming  there  are  no  other  statements  assigning  values  to  A): 

float  array  A(0. .n,0. .n) ; 
if  i=0  or  j*0  then  A:»el; 
for  i:*  2  to  n  do  { 
for  j  :»  2,  n  do  •( 

A(i,j)  :=  A(i-l,j)  +  A(i,j-1)  }  } 

Let  V  be  the  data  type  of  floating  point  numbers,  in  the  notation  of  data  fields,  the  above 
is  written  as 


Do  =  interval(0,n)  D  —  Do  "X  Dq 

a:D- V  =  1 

[else  a(t  -  1,;)  -f  a{i,j  -  1)  J 

The  primary  role  of  index  of  domain  morphisms  is  in  defining  new  data  fields. 


Definition 

mapping 


A  data  field  morphism  induced  by  an  index  domain  morphism  g  :  D  —*  E,  h  & 
g‘  :{E -^V)^{D -*V):a^aog 


where  D  V  and  E  —*V  are  sets  of  data  fields. 


Given  a  data  field  a  :  D  V  and  a  domain  morphism  g  :  D  E,  what  we  generally 
want  is  to  find  the  new  data  field  d  such  that  g*{a)  =  o.  In  order  to  solve  this  equation  we 
need  the  inverse  of  g,  i.e.  g  needs  to  be  a  reshape  morphism.  Then  given  g  and  g~^,  we  can 
formally  derive  a  =  5r“^*(a)  =  ao  g~^. 


2.5  Constructs  for  other  Semantic  Entities 

Since  a  compiler  relies  on  syntactic  cues  to  do  its  optimizations,  we  found  it  important  to 
make  explicit  certain  semantic  entities  which  are  implicit  in  current  programming  languages, 
hyper-surfaces  and  communication  forms. 
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Hyper-Surfaces  Hyper-surfaces  define  the  boundaries  of  domains  where  functions  take 
on  special  values.  The  idea  is  to  elevate  boundaries  cf  domains  to  be  semantic  entities  and 
promote  a  programming  style  where  the  same  semantic  entity  has  a  single  syntactic  entity 
corresponding  to  it.  This  way,  the  repetitive  occurrence  of  related  boolean  expressions  for 
testing  a  particular  boundary  often  seen  in  programs  can  be  eliminated.  The  notion  of 
hyper-surface  makes  the  algebraic  program  transformation  easier  and  more  elegant,  as  well 
as  helps  the  compiler  recognize  domain  boundaries  and  do-  optimizations  such  as  aligning 
multiple  data  structures  and  reducing  storage  use. 

In  this  paper,  the  notion  of  hyper-surfaces  is  used  only  in  the  example  of  Section  ??,  we 
therefore  give  the  definitions  in  Section  ??  of  the  Appendix. 

Communication  Forms  Given  an  index  domain  D,  a  communication  form,  7,  is  a  com¬ 
position  of  communication  operators.^  The  idea  is  that  given  an  index  (considered  as  the 
receiver  of  a  message),  the  communication  form  produces  the  “sending”  index  of  the  message. 
Conversely,  the  left  inverse,  if  it  exists,*  of  a  communication  form  indicates  the  “receiving” 
index  given  an  index  interpreted  as  the  sender  of  a  message. 

An  asynchronous,  distributed-memory  parallel  machine,  at  its  lowest  level  of  communica¬ 
tion  system,  requires  to  know  both  the  sender  and  receiver  in  order  to  establish  synchroniza¬ 
tion  and  communication.^  For  the  compiler  of  any  parallel  language  without  explicit  “send” 
and  “receive”  commands,  communication  forms  must  be  extracted  and  its  inverse  derived  or 
provided  by  user  in  the  form  of  directives. 

In  the  following,  we  show  the  effect  of  domain  morphisms  on  communication  forms,  which 
provides  vital  information  for  generating  communication  commands  in  the  target  code. 

2.6  Effect  of  Reshape  Morphism  on  Communication  Form 

Consider  a  data  field  a  :  D  —*  V  defined  by 

which  contains  a  call  to  data  field  b:  D  —*  V  using  the  communication  form  7  on  index  i. 
Let  g  :  D  —*  E[P  x  Ad]  be  a  reshape  morphism  from  D  to  E  with  inverse  The  resulting 
data  fields  after  the  reshape  transformation  are  a  —  a  o  g~^  zmd  6  =  60  g~^ .  Suppose 
7  is  bijective  and  7"*  given, ^  then  the  new  communication  form  and  its  inverse  after  the 
transformation  are 

7  =  5'0  7  05r”*  and  7“^  =  ^ o 7"*  o 

as  shown  in  the  commuting  diagram  of  Figure  ??. 

The  formulation  of  the  new  communication  form  shows  that  in  compiler  design  and  paral¬ 
lel  language  design,  we  should  do  the  following:  (1)  By  all  means,  try  to  obtain  closed  forms 
for  7  and  its  inverse  at  compile  time  so  that  they  do  not  have  to  be  evaluated  at  runtime.  (2) 

'This  is  only  a  special  type  known  as  a  simple  communication  form.  In  general,  a  communication  form 
can  be  defined  to  be  a  set  of  relations  representing  multiple  many-to-many  communications. 

^Again,  in  general,  we  consider  the  inverse  relation,  which  always  exists. 

^Please  refer  to  Hoare’s  CSP  model  [?]. 

^Recall  that,  in  general,  7  is  a  set  of  relations  and  the  result  generalizes  accordingly. 
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go^~^  og 


Figure  1:  Diagram  showing  the  new  communication  form  induced  by  domain  morphism  g 
and  communication  form  7. 

If  the  above  is  not  possible,  devise  am  eificient  run-time  implementation  to  evaluate  7  and 
its  inverse.  (3)  The  information  needed  to  do  the  above  is  contained  in  the  original  commu¬ 
nication  forms  (and  their  inverses)  and  the  domain  morphism  (and  its  inverse);  hence,  the 
compiler  must  generate  the  morphisms  and  the  communication  forms  either  automatically 
or  rely  on  user  provided  directives. 

3  Program  Transformation 

Data  fields  and  domain  morphismsare  semantic  entities  which  are  represented  in  a  program¬ 
ming  language.  Semantically,  a  new  data  field  can  be  defined  as  the  composition  of  a  data 
field  and  a  domain  morphism.  In  this  section  we  describe  an  equational  theory  that  2dlows 
the  representation  of  the  new  data  field  to  be  obt^lined  by  manipulating  the  representations 
of  the  given  data  field  and  domain  morphism.  First,  we  introduce  the  equational  theory,  and 
then  describe  the  steps  in  the  program  transformation  using  reshape  morphisms. 

3.1  Equational  Theory 

By  an  equational  theory  of  a  programming  language  we  mean  a  set  of  valid  equations  or 
algebraic  identities  in  the  language  {M  =  N)  along  with  inference  rules  for  deriving  new 
equations  from  old.  It  is  based  on  the  equational  theory  of  Church’s  lambda  calculus  [?]. 
The  a  and  0  rules  are  well  understood,  but  it  turns  out  that  17-abstraction  is  essential  in 
order  to  simplify  certain  expressions.  Figure  ??  lists  a  minimal  set  of  algebraic  identities  for 
a  functional  language  with  function-abstraction,  the  conditional  expression  and  comi>osition 
of  functions.  Some  of  these  identities  are  listed  in  Backus’s  work  on  FP  [?],  though  the  key 
difference  is  that  he  does  not  have  any  involving  explicit  function-abstraction. 

Figure  ??  lists  the  basic  inference  rules  including  the  usual  ones  for  equality  (reflex- 
ivity,  symmetry,  and  transitivity),  substitution,  as  well  as  ones  obtained  from  application, 
abstraction  and  composition. 

A  definition,  in  an  equational  theory,  is  a  simple  equation  where  the  left  hand  side  is 
a  single  variable  and  the  right  hsmd  side  is  some  expression  which  may  contain  the  same 
variable  (recursive  definitions).  In  the  framework  of  [?],  a  definition  enriches  a  theory  with 
a  new  operator  and  a  new  equation.  For  a  mutually  recursive  set  of  function  definitions,  we 
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Hx)-.T{M]  =  fn(!,):r{Mli/!,l}  (q) 

fn(i):r{Jl/}(A')  =  M[x/N]  {0) 

fn(x):7’{A/(i)}  =  M  (i  not  free  in  Af)  (./) 

Figure  2:  Algebraic  identities  from  the  A-calcuIus,  where  M  and  N  are  expressions  and  T  is 
a  type-expression. 


Fo{b-^  h]  = 
{b-^  h}oF  = 
= 


{B-tFoH} 

{BoF  ->  H  oF] 

{fn(x)(B(i))-,f„(x){ff(x))} 


(l-dist-comp-if) 

(r-dist-comp-if) 

(dist-app-if) 

(dist-abs-if) 


Figure  3:  Algebia’c  identities  involving  the  conditional,  where  F  and  H  are  function  expres¬ 
sions,  and  fi  is  a  boolean  function  expression. 

A/f  —  h/f  M  —  N  M  —  L  and  L  =  N 

^  =  M  - 

M  =  N  M  =  N 

M  o  F  =  N  o  F  H  o  M  =  iJ  0  N 

M=N  M=N  M=N 

=  fnix){N}  m(L)  =  N{L)  L[M)  =  L{N) 

M  =  N 

M[H/K\  =  N[H/K\ 

Figure  4:  Inference  rules,  where  M[HfK]  denotes  the  new  term  with  H  replaced  by  K,  the 
substitution  operation  of  the  A-calculus. 


consider  the  functions  that  are  implicitly  defined  as  satisfying  all  the  defining  equations  in 
the  enriched  theory. 


3.2  Strategy  for  Obtaining  New  Definitions 

With  the  equational  theory  providing  the  algebraic  identities  and  the  inference  rules,  we 
describe  the  strategy  of  formally  transform  the  original  program  into  a  more  efficient  one. 
For  simplicity  we  begin  with  a  program  consisting  of  one  definition: 

a  =  fn(i):i:?{Ti[a]}, 

where  Ti[a]  is  an  expression  in  i  possibly  containing  a.  By  an  abuse  of  notation,  we  also  use 
a  to  denote  the  data  field  defined.  Next,  let  the  reshape  morphism  g  and  its  inverse  be  given 
by 

g  =in{x):D{T2:E}  and  g~^  =  in{y):E{T3:D}. 

Semantically,  what  we  want  is  a  data  field  a  satisfying  a  =  aog~^.  However,  merely  executing 
the  program  g~^  followed  by  a  does  not  decrease  the  communication  cost.  What  we  waint  is 
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a  new  definition  of  a  which  does  not  contain  either  a,  g  ot  g  A  strategy  for  obtaining  a 
new  definition  for  a  from  the  definitions  of  a,  g  and  g~^  is  the  following: 

1.  Using  the  identity  a  =:  aog,  replace  all  occurrences  of  a  with  aog  in  the  definition  of  a. 

2.  Using  a  combination  of  unfoldings  of  g  and  g~^  and  various  other  identities  given  in  the 
theory,  eliminate  all  occurrences  of  g  and  g~^  from  the  result  of  the  first  step.  A  very 
useful  transformation  turns  out  to  be  the  abstraction,  where  we  provide  a  function 
with  dummy  arguments  in  order  to  unfold  it. 

3.3  Metalanguage 

Since  the  equational  theory  deals  with  programs,  and  equations,  a  metalanguage  for  manip¬ 
ulating  program  expressions  can  be  defined  [?].  The  derivation  strategy  described  above  can 
be  programmed  in  the  metalanguage  using  metalanguage  operators,  the  constructors  and 
selectors. 

In  the  following  example  derivation,  the  metalanguage  operators  that  achieve  the  trans¬ 
formation  appear  in  square  brackets. 


4  Example:  Successive  Over-Relaxation 

We  illustrate  our  program  transformation  strategy  on  the  successive  over-relaxation  defined 
in  Program-1.  This  program  solves  Laplace’s  equation  using  finite  difference  formula¬ 
tion.  The  program  finds  solutions  for  the  unknown  potential  within  a  finite,  discretized  2- 
dimensional  spatial  domain  (defined  by  index  domain  Do  x  Do)  with  fixed  boundary  values 
and  initial  values  (conditional  branches  guarded  by  hyper-surfaces  So  and  Si,  respectively). 
The  algorithm  is  iterative  (over  index  domain  Di),  and  in  each  iteration  the  potential  at 
every  point  of  the  2-dimensional  grid  is  the  average  of  the  values  of  its  four  neighboring 
points  and  itself.  The  over-relaxation  ordering  is  such  that  the  values  of  the  current  itera¬ 
tion  are  used  for  the  upper  ((i  —  1,  j))  and  left  ((i  —  1,  j))  neighbors,  while  those  from  the 
previous  iterations  are  used  for  itself  {{i,j))  and  the  other  two  neighbors.  This  algorithm 
uses  0{‘n?)  space  but  only  0{n)  parallelism  due  to  the  over-relaxation  ordering.  The  point 
of  the  domain  morphism  is  to  transform  this  program  to  another  that  has  0{n^)  parallelism. 
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Program- 1 

!  Index  Domains 

Dq  =  interval(0,n)  Di  =  interval(l,m)  D  =  prod-dom(J[)o,Do,-Di) 
!  Hyper-Surfaces 

Sq  =  in{iJ,k):D{{i  =  0,  >)  A  {i  =  n,<)  A  {j  =  0,  >)  A  {j  =  n,<)} 

=  1,>)  A  (A;  =  m,<)} 

!  Data  Field 

a  =  in{i,j,k):D 


O  O 

T 

o 

Si{iJ,k)  AoiiJ) 

•> 

SHi,j,k)  - 

Stii,j,k)  f{a{i-  l,j,k),a{i,j  -  l,jfc). 

► 

a{i,j,k-  l),a{i  +  l,j,k-  1), 

k 

a{i,j +  l,k  -  1)) 

J 

Although  the  program  uses  hyper-surfaces  in  the  guards  (see  Section  ??  for  detailed  descrip¬ 
tions),  for  comparison,  we  present  the  equivalent  boolean  expressions: 

=  (*  =  0  V  i  =  n  V  j  =  0  V  j  =  n)  A  (0  <  i  <  n)  A  (0  <  j  <  n)  A  (1  <  fc  <  m) 

Soihj)^)  =  (0  <  i  <  n)  A  (0  <  j  <  n)  A  (1  <  /:  <  m) 

S°{i,j,  k)  =  (fc  =  1  V  fc  =  m)  A  (0  <  i  <  n)  A  (0  <  j  <  n)  A  (1  <  fc  <  m) 

k)  =  (0  <  i  <  n)  A  (0  <  j  <  n)  A  (1  <  <  m) 

The  use  of  hyper-surfaces  not  only  allows  the  compiler  to  extract  useful  optimization  infor¬ 
mation  from  the  program,  but  also  provides  conceptual  tools  for  reasoning  about  the  different 
boundaries. 

i 

I 

4.1  New  Domains  and  Domain  Morphisms 

First,  we  define  the  new  domain  E  and  domain  morphisms  g  \  D  —*  E  and  g~^  •.  E  —*  D.  The 
kind  of  domain  E  indicates  that  the  first  two  coordinates  are  to  be  interpreted  as  spatial, 
and  the  third  temporal. 

Morphisms 

E  =  image(5',  D)  where{  g'  =  fn(i,;,  k):D{{i,j,  i+j  +  2*  k)}  } 
g  =  iT\{i,j,k):D{{i,j,i  +  j  -\-2*  k):E[SxSxT]} 

9~'  =  ^n(i,j,fc):E{(i,i,(A:-t-j)/2):D}. 


Next,  define  new  surfaces  Rq  and  R\  which  satisfy: 

Rq  =  Sqo  g~^  Ri  =  Si  o  g~^ 

So  =  RqO  g  Si  =  Ri  o  g 
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4.2  Hyper-Surface  Derivations 

Ri  =  Si  og-^ 

=  fn(i,  j,  k):E{Si{i,j,  (k  —  i  —  i)/2)}  (Unfold  composition  aind  5'“^) 

=  fn(i,  j,  k):E{ir\{i,j,  k):D{{k  =  1,  >)  a  (fc  =  m,  {k  -  i  -  j)/2)} 

=  fn(ui,  A:) :  JS{((fc  -  i  -  j)/2  =  1,  >)  A  {{k  -  i  -  j)/2  =  m,  <)} 

=  fn(i,  k):E{{k  —  i— j=2,>)A{k  —  i— j  =  2*m,  <))} 

Similarly,  we  obtain 

Ro  =  in{i,j,k):E{{i  =  0,>)  A  (i  =  n,  <)  A  (j  =  0,  >)  A  {j  =  n,<)} 
which  has  the  same  body  as  So  since  there  are  no  constraints  on  the  third  coordinate  (k). 

4.3  Data  Field  Derivation 

Let  /Co  denote  the  equation  defining  a  in  Program-1.  The  derivation  of  the  definition  for 
the  new  data  field  a  makes  use  of  the  identities 

a  =  a  o  g~^  and  a  =  ao  g. 

(Note:  we  adopt  the  convention  that  composition  binds  more  tightly  than  application,  so 
that  f  og{x)  =  f(g(x)).) 

1.  Substitute  V  with  ’’a  o  g''  in  «o  [«i  =  subst(/co,  V,  o  g"')]: 

ao  g  =  fn(z,  j,  k):D 

-*  Ao{i,j) 

SHhhk)  f{aog{i  -  l,j,k),aog{i,j  -  1,  A:), 

aog{i,j,k-  1),  a  o  ^(i -f  1,  j.  A:  -  1), 
aog{i,j  -f  1,A:  -  1)) 

2.  Right-compose  both  sides  of  /Ci  with  and  then  simplify  compositions  by  eliminat¬ 

ing  identities  {g  o  g~^  t>  and  g~^  o  g>  l^)  and  unfold  the  composition  operator  when 
possible  (/  o  ^(i)  o  f{g{x))) 

[/C2  =  simplify-comp(right-comp(«i, 

a  =  [fn(i,j,A:):D 

•« 

S°{i,j,k) Ao{iJ) 

Siii,j,k)  f{a{g{i-l,j,k)),a{g{i,j  -  l,k)), 

ai9{ij,k  -  l)),a{g{i  +  lj,k  -  1)), 

+  l,k-l)))  J, 
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3.  Distribute  the  abstraction  k):D^  over  the  conditional  [ks  =  dist-abs-if(«;2,  '^(i,  J,  k) 
D-')]: 

fn(i,  j,  k):D{S^{i,j,  A:)}  -f  fn(i,  j,  A:):D{0} 

in{iJ,k):D{S^{i,j,k)}  in{i,j,k):D{Ao{i,j)} 

in{i,j,k):D{St{i,j,k)} -i 
a  =  <  >  ^  9 

in{i,j,k):D{ 

f{a{g{i  -  lj,k)),a{g{i,j  -  1,A:)), 
ai9iij,k-  l)),a{g{i  +  l,j,k-  1)), 
a(5(*,i  +  1,A:- 1)))} 


4.  Perform  7y-reductions  (fn(a:):D{M(x)}  >  M)  [k4  =  eta-reduce(/C3)]: 


[  fn(i,j,/c):D{0} 

St  ^ir\{i,j,k):D{ 

Sq  — +  < 

f{a{g{i  -  l,j,k)),a{g{i,j  -  l,fc)), 

► 

Hgihj,  ^  -  l)).a(5(*  + 1,;,  - 1))) 

%(*,;■  + 1,^- 1)))} 

< 

5.  Right-distribute  the  composition  over  the  conditional  [«5  =  r-dist-comp-if(/C4)]: 


[5^05  * (fn(i,j,A:):D{0})  0^  ^ 

S°og~^  (fn(z,j,A:):  D{Ao(i,j)})  05-^ 

St  °9~^  (fn(i,i,A;):D{ 

So  0  5”*  -♦  < 

f{a{9{i  -  lj,k)),a{g{i,j  -  1,  A:)), 

► 

a(S'(*,i,A:  -  l)),d(sf(i  +  1,;,  A:  -  1)), 

1. 

+ 1.*;- l)))})o5"^ 

4 

6.  Substitute  o  g  with  and  *^5^  o  ”  with 
[k6  =  subst(subst(/C5,  o  flr-”,  '’5J’  o  g-^'', 


d 


{in{i,j,k):D{0})og  ^ 

R+-.(fn(,-,i,A:):D{ 

l)))a(5(*  +  -  1))> 

a{9{iJ  +  hk-l)))})og-‘^ 

4 
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7.  Eta-abstract  the  right-hand  side  of  the  equation  by  \i,j,  k):E'' 

[kt  =  mk-eqn('^d'',  mk-eta('^(i,  j,  rhs(K;6)))]: 
d  =  in{i,j,k):E{ 

-R?  {in{iJ,k):D{AQ{i,j)})og-^ 

^  R+->(fn(i,j, /:):!>{  >(i  j  k)} 

^ '  f{a{9{i-l,j,k)),a{g{i,j -l,k)), 

a{9{i,j,k~l)),a{g{i  +  l,j,k-  1)), 
l,k- l)))))og-^  J 

8.  Distribute  application  over  conditional  [/cg  =  dist-app-if(/C7)]: 

a  =  fn(iJ,k):E{ 

(fn(t,j,k):I){0})  og-^(tJ,k) 

->■  (fn(hJ,  k):E{Ao(t,J)})  o  g-\ij,  k) 

'  D+c  ■  L\ 

fia{g{i  -  l,j,k)),a{g{i,j  -  l,fc)), 

l)),a(5(*  +  -  1)), 

a{giij  +  l,k-  1)))})  o  g-'^{i,j,  k) 


9.  Simplify  composition  [xg  =  simplify-comp(Kg)]; 
a  =  in{i,j,k):E{ 

k):D{Aoii,j)})i9~'^{iJ,k)) 

Rtihj,  k) 

'  ji+(i  ■  k)-*< 

/(%(*- -  1,*:)), 

-  l)).a(sr(i  -t-  -  1)), 

Kgihj  + 1,  fc  -  i)))})(y‘M^i»  *)) 


10.  Unfold  •’flr"”  and  V  [kiq  =  unfold(unfold(K9,  V  ”)>  V)]‘ 


a  =  fn(i,;,fc):E{ 
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(^n(i,  j,  k):D{0}){i,j,  {k-i-  j)/2) 
k)  -4  k):D{Ao(i,j)})(iJ,  (k-i-  j)/2) 

(fn(i,j,k):D{ 

f{a(i  -  l,j,i  -  1  +  j  +  2*  k),a{i,j  -  1,  i  -t- ji  -  1  -|-  2  *  fc), 
j,*'  -t-j  +  2  ♦  A:  —  2),a(t  -I- 1,  j,t  2  *  A;  -  1), 
aii,j  +  l,i+j  +  2*k-l))}){i,j,(k-i-j)/2) 

11.  Perform  /3-reduction  and  simplify  the  arithmetic  [kh  =  simlify-arith(reduce(«io))]: 
a  =  in{i,j,k):E{ 

R^(iJ,k)  0 

RlihJ,  k)  Ao{iJ) 

fi^ii  -  -  l),a(ij  -  1,A:  -  1), 

a(i,j,A:-2),a(i-l-l,j,A:-l), 

a(i,j -I- l,fc  -  1)) 


} 


Ro(hj,k) 


} 


The  resulting  index  domain  and  data  field  definitions  appear  in  Prograni-2.  The  program 
captures  the  red/black  checkerboard  algorithm  of  SOR  described  in  Chapter  7  of  [?],  except 
for  the  final  partitioning  of  the  two  spatial  coordinates  onto  processors  and  memory,  which 
again  can  be  represented  as  a  domain  morphism  and  the  transformation  performed  with  the 
same  procedure  but  with  difference  in  the  detail  in  arithmetic  simplification. 


Program-2 
!  Index  Domain 

E  =  image(y,D)  where{  /  =  A:):D{(i,;,  z -|- j  +  2  *  A:)}  } 

!  Hyper-Surfaces 

/lo  =  fn(z,;,A:):f;{(z  =  0,>)  A(z  =n,<)A(;  =0,>)  A(;  =n,<)} 
Ri  =  in{i,j,k):E{{k-i-  j  =  2,>)  A(k  -  i  -  j  =  2*m,<))} 

!  Data  Field 

a  =  fn(z,  j,  A:):£' 


R^{hj,k)  -»  0 

Ii°i{iJ,k)  Ao{i,j) 

' 

H^(iJ,k)  -4  ^ 

Rtihj,k)  -4  /(a(i-  l,j,fc-  l),a(z,j  -  1,A:-  1), 

a(z,;,A:-2),a(z-f  l,i,A:-l), 

a(i,j  -h  l,k  -  1)) 

We  translate  Program-2  into  an  imperative  program  to  illustrate  the  effect  of  the  domzun 
morphism  on  the  implementation.  We  begin  with  the  data  structure.  By  analyzing  data 
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dependence  of  a,  we  know  that  a  two  dimensional  array  is  needed  to  store  the  values  of  a  at 
each  point  of  the  spatial  component  of  domain  E.  The  recursive  call  made  to  k  —  2) 

indicates  a  value  of  2  time  step  ago  is  needed.  Since  each  element  of  a  ‘3  changed  only  once 
every  two  time  steps,  there  is  no  additional  storage  needed. 

Next,  the  control  structure.  The  bounds  of  the  for- loop  is  derived  from  the  temporal 
component  (indexed  by  k)  of  domain  E,  while  those  of  the  forall-loop  from  the  spatial 
component  of  E.  The  red/black  checkerboard  ordering  of  the  over- relaxation  is  controlled  by 
the  predicates  of  the  if-statement  in  the  forall-loop.  These  predicates  are  derived  from  the 
hyper-surface  Ri  and  the  domain  E.  In  particular,  the  test  whole?  ((k-i-j)/2)  is  to  ensure 
only  those  elements  of  E  that  are  images  of  elements  of  D  (which  are  integral)  participate 
in  the  computation.  Note  that  because  of  the  limitation  of  the  syntax  for  specifying  loop 
bounds  in  current  programming  languages,  the  domain  information  must  be  specified  both 
in  the  loop  bounds  and  in  the  if-statement. 

Note  that  Program-2  contains  all  the  necessary  information  for  a  translator  to  generate 
the  imperative  program. 

function  SOR(AO) 

float  array  A(0. ,n,0, ,n) ; 

A(i,j)  :=  A0(i,j); 
for  k:=  2  to  2*(n+in)  do  { 
forall  (i,j)  in  (0..n  ,  0..n)  do  { 
if  (2*in  >  k-i-j)  and  (k-i-j  >  2)  and  whole? ((k-i-j)/2) 
then  A(i,j)  :=  f(A(i-l,j),  A(i,j-1),  A(i,j),  A(i+l,j),  A(i,j+1)) 
fi  }  } 

5  Concluding  Remarks 

The  complexity  of  managing  explicit  parallelism  to  produce  efficient  code  remains  a  major 
obstacle  in  the  widespread  use  of  the  new  generation  of  parallel  machines.  At  the  same  time, 
the  task  of  automatically  generating  parallel  programs  appears  to  be  extremely  difficult  eind 
costly,  except  for  a  very  restricted  class  of  problems.  Recognizing  these  issues,  we  know  that  a 
balance  must  be  sought  between  the  automatable  and  the  effort  required  of  the  programmer. 

Our  work  provides  a  theoretical  basis  for  and  makes  evident  where  to  draw  the  line 
between  what  is  automatable  and  what  needs  user-directives.  As  far  as  we  know,  this  is 
a  first  attempt  at  formalizing  parallelizing  compiler  techniques.  One  of  its  consequence  is 
to  provide  a  framework  for  systematic  application  of  these  techniques,  where  a  reference 
metric  is  defined  over  the  communication  forms  to  provide  guidance  in  minimizing  certain 
cost.  Parallelization  and  optimization  techniques  have,  by  and  large,  been  done  in  an  ad  hoc 
fashion,  and  we  have  just  began  to  understand  their  interactions  now  that  this  theoretical 
framework  is  in  place. 
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A  Hyper-Surfaces 

Hyper-surfaces  define  the  boundaries  of  domains  where  functions  taJce  on  special  values.  By 
making  hyper- surfaces  first  class  objects,  we  eliminate  the  repetitive  occurrence  of  related 
boolean  expressions  for  boundary  testing  which  makes  compiler  optimization  difficult. 

A  data  field  definition  will  usually  contain  a  conditional  expression  which  tests  for  certain 
boundary  conditions  from  the  interior  of  the  domain  of  definition.  These  tests  are  usually 
expressed  as  inequalities  over  the  formal  parameters.  When  many  data  fields  are  defined 
over  the  same  domain,  the  tests  may  be  combined  and  transformed  into  very  different  forms, 
making  the  domain  analysis  very  difficult. 

In  order  to  preserve  the  semantic  content  of  tests  for  boundary  conditions,  we  introduce 
a  general  notion  of  a  hyper-surface  which  specifies  the  boundary  of  a  domain,  and  a  number 
of  operators  for  testing  whether  a  point  is  on  the  positive  or  the  negative  “side”  of  the 
boundary,  or  on  the  boundary. 

In  the  following,  all  manifolds  are  assumed  to  be  simply  connected,  such  as  Euclidean 
spaces.  Note,  however,  that  the  notion  of  hyper-surface  can  be  extended  to  trees  and  other 
discrete  structures  which  contain  a  notion  of  locality  and  can  be  partitioned  into  the  positive 
and  the  negative  components. 

Definition  For  any  n-dimensional  manifold  M,  a  hyper-surface  is  an  (n  —  l)-dimensional 
sub-manifold  S  which  partitions  M  into  two  components  known  as  the  positive  and 

the  negative  (5j^)  components. 

For  each  surface  S  in  M,  we  define  three  associated  boolean  valued  functions.  For  each 
point  X  in  M: 

5°(x)  iff  X  G  5  (surface) 

5''’(x)  iff  X  €  (positive-component) 

5“(x)  iff  X  G  S]^  (negative-component) 

By  an  abuse  of  notation,  we  write  5(x)  for  S°(x).  Usually,  we  just  write  S'*"  when  the 
ambient  manifold  M  is  understood. 

A.l  Representation 

For  n-dimensional  Euclidean  space  over  the  reals,  a  hyper-surface  can  be  specified  by  an 
equation  t[ii,...,x„]  =  0,  where  the  left  hand  side  contains  at  least  one  index  x,  with 
non-zero  coefficient,  and  an  orientation.  One  way  to  specify  the  positive  component  with 
respect  to  the  hyper-surface  is  by  modifying  the  equation  into  an  inequation.  There  are  two 
possibilities,  the  positive  component  can  be  defined  as  the  set  of  points  which  satisfy  either 
T  <  0  or  T  >  0.  We  formalize  this  into  a  hyper-surface  constructor. 

Definition  Let  D  be  an  n-dimensional  manifold.  The  expression 

fn(xi,...,x„)  :  D{(t[xi,...,x„]  =  0,7)} 

denotes  a  simple  hyper-surface  defined  by  the  equation  t[xi,...,x„]  =  0  and  the  positive 
component  is  given  by  7,  which  can  be  either  <  or  >. 
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Example  Let  D  be  the  3-dimensional  Euclidean  space.  Then 

k) :  D{{k  -{i+  j)/2  =  0,  >)} 

denotes  a  hyper-surface  of  D  such  that  A:)  iff  fc  —  (i  -I-  j)/2  =  0  and  S'^{i,j,k)  iff 

k  —  {i  +  j)/2  >  0. 

A. 2  Hyper-Surface  Operations 

Let  Si  and  S2  be  hyper-surfaces  of  M.  We  can  define  the  conjunction  {Si  A  S2),  negation 
(Si),  disjunction  (Si  V  Sj)  using  the  notion  of  positive  component: 

(Si  A  S2)'*’  =  Sj*"  n  S^  and  S^  =  S~  and  Si  V  S2  =  Si  A  S2. 

The  effect  of  conjunction  is  to  take  the  conjunction  of  the  two  defining  equations,  while  that 
of  negation  is  to  switch  <  and  >. 

In  general,  we  can  define  complex  hyper-surfaces  by  allowing  logical  conjunction,  disjunc¬ 
tion,  and  negation  in  the  body  of  the  hyper-surface  constructor: 

fn(x)  :  D{{Ti[x]  =  0,71)  A  {t2[x]  =  0,72)} 

=  (fn(i)  :  D{{ti[x]  =  0,71)})  A  (fn(a:) :  D{{t2[x]  =  0,72)}) 

fn(x)  :  D{(ri[x]  =  0,7i)  V  (t2[x]  =  0,72)} 

=  (fn(x)  :  L>{(ri[x]  =  0,7i)})  V  (fn(x)  :  L>{(t2[x]  =  0,72)}) 


fn(x)  :  D{-'(t[x]  =  0,7)}  =  fn(x)  :  D{{t[x]  =  0,7)}  =  fn(x)  :  D{{t{x]  =  0,7')} 
where  7'  is  one  of  {<,  >}  which  is  the  opposite  of  7. 


